Spatialized haptic feedback based on dynamically scaled values

ABSTRACT

A system provides haptic feedback based on media content. The system processes the media content into components including a first component and a second component. The system further determines a first priority value related to the first component and a second priority value related to the second component. The system further compares the first priority value with the second priority value. The system further generates a first control signal and a second control signal based on the comparison, where the first control signal is configured to cause a first haptic feedback to be output and the second control signal is configured to cause a second haptic feedback to be output that is the same or different than the first haptic feedback.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority of U.S. Provisional Patent ApplicationSer. No. 61/874,678, filed on Sep. 6, 2013 (the disclosure of which ishereby incorporated by reference).

FIELD

One embodiment is directed generally to a device, and more particularly,to a device that produces haptic effects.

BACKGROUND

Electronic device manufacturers strive to produce a rich interface forusers. Conventional devices use visual and auditory cues to providefeedback to a user. In some interface devices, kinesthetic feedback(such as active and resistive force feedback) and/or tactile feedback(such as vibration, texture, and heat) is also provided to the user,more generally known collectively as “haptic feedback” or “hapticeffects”. Haptic feedback can provide cues that enhance and simplify theuser interface. Specifically, vibration effects, or vibrotactile hapticeffects, may be useful in providing cues to users of electronic devicesto alert the user to specific events, or provide realistic feedback tocreate greater sensory immersion within a simulated or virtualenvironment.

Dynamically providing haptic feedback based on media components of mediacontent can provide a powerful and immersive experience for the user.However, in some instances, the different media components can havemedia characteristics that are different but sufficiently similar suchthat haptic feedbacks dynamically generated based on the mediacomponents are difficult to distinguish. For example, in a stereo audioimplementation, haptic feedback corresponding to the “right” audiochannel and provided via a “right” haptic feedback output device may beindistinguishable from haptic feedback corresponding to the “left” audiochannel and provided via a “left” haptic feedback output device whenaudio for the right and left audio channels are sufficiently similar. Inthe foregoing example, the user may not perceive a difference betweenthe haptic feedbacks corresponding to the right and left channels.

SUMMARY

One embodiment is a system that provides haptic feedback based on mediacontent. The system processes the media content into componentsincluding a first component and a second component. The system furtherdetermines a first priority value related to the first component and asecond priority value related to the second component. The systemfurther compares the first priority value with the second priorityvalue. The system further generates a first control signal and a secondcontrol signal based on the comparison, where the first control signalis configured to cause a first haptic feedback to be output and thesecond control signal is configured to cause a second haptic feedback tobe output that is the same or different than the first haptic feedback.

BRIEF DESCRIPTION OF THE DRAWINGS

Further embodiments, details, advantages, and modifications will becomeapparent from the following detailed description of the preferredembodiments, which is to be taken in conjunction with the accompanyingdrawings.

FIG. 1 illustrates a system for providing haptic feedback based ondynamically scaled priority values of media components of media content,according to an embodiment of the invention.

FIG. 2 is a flow diagram illustrating data flows between components in asystem for providing haptic feedback based on dynamically scaledpriority values of media components of media content, according to anembodiment of the invention.

FIG. 3A illustrates an example of a computing device for prioritizingaudio channels with integrated haptic and audio output devices,according to an embodiment of the invention.

FIG. 3B illustrates an example of a computing device for prioritizingimage components with integrated haptic and image output devices,according to an embodiment of the invention.

FIG. 4 illustrates an example of a computing device for prioritizingaudio channels with remote haptic and audio output devices, according toan embodiment of the invention.

FIG. 5A is an example of a graph that illustrates scaling priorityvalues, according to an embodiment of the invention.

FIG. 5B is an example of a graph that illustrates scaling priorityvalues, according to an embodiment of the invention.

FIG. 5C is an example of a graph that illustrates scaling priorityvalues, according to an embodiment of the invention.

FIG. 5D is an example of a graph that illustrates scaling priorityvalues, according to an embodiment of the invention.

FIG. 6 illustrates an example of a process for providing haptic feedbackbased on dynamically scaled priority values of media components of mediacontent, according to an embodiment of the invention.

FIG. 7 illustrates an example of scaling priority values, according toan embodiment of the invention.

FIG. 8 illustrates an example of scaling priority values, according toan embodiment of the invention.

FIG. 9 illustrates an example of a process for providing haptic feedbackbased on input components of media content, according to an embodimentof the invention.

DETAILED DESCRIPTION

One embodiment is a system that provides haptic feedback that variesspatially based on dynamically scaled values of media components, suchas audio channels, regions of an image, or other media components ofmedia content. The system may analyze the media components andautomatically generate haptic feedback based on the analysis. The systemmay provide the haptic feedback via haptic output devices incoordination with output of the media components via speakers, imagedisplays, or other media output devices. Thus, a given media componentmay be output along with its corresponding haptic feedback, which can bedetermined dynamically by the system.

In order to emphasize certain haptic feedback that is based on certainmedia components over other haptic feedback that is based on other mediacomponents, the system may scale one or more portions of the hapticfeedback based on an analysis of the media components. For example, thesystem may scale information that causes haptic feedback to begenerated, causing certain haptic feedback to be more pronounced (e.g.,increased magnitude, duration, etc.) and/or causes other haptic feedbackto be less pronounced (e.g., decreased magnitude, duration, etc.). Thesystem may scale the information such that the original media componentsare unaltered when played back by image, audio, and/or other outputcomponents. In other words, the original media content may be playedback unaltered while certain haptic feedback automatically generatedfrom the media content is more or less pronounced than other hapticfeedback (although in some implementations, the media content may beseparately and/or together scaled with the haptic feedback as well). Thesystem therefore facilitates dynamic haptic feedbacks that can bedistinguished by the user even when such haptic feedbacks are based onmedia components that are similar to one another.

In this manner, the system can improve the ability to distinguishbetween different haptic feedback that is provided in association withdifferent media components and can provide a richer, more immersivemedia experience. Such enhancements can be applied to audio, images(such as video or still images), video games, and/or other mediacontent.

In one embodiment, the system may prioritize the media components andgenerate a control signal that causes haptic output devices to providehaptic feedback based on the prioritization. In some implementations,for example, the system may process media content into different mediacomponents and analyze each of the media components to automaticallydetermine a corresponding priority value used to dynamically determinehaptic feedback. The analysis may be based on one or more mediaproperties of a given media component. The media properties may include,for example, a magnitude of sound (among other audio properties) foraudio content, a color (among other image properties) for image content,and/or other properties of media content that can be quantified orotherwise evaluated for analysis.

The system may compare each of the priority values of media componentswith a priority value (or aggregate value, such as an average) of atleast one other media component. If a difference between the priorityvalues exceeds a threshold value, the priority value may be scaled up ordown. Thus, some priority values may be scaled up, others may be scaleddown, while still others may not be scaled at all. In this manner, thehaptic feedback that is generated based on the priority values for mediacomponents may be more distinguishable due to the scaling. In someimplementations, the system may scale the priority values irrespectiveof whether a threshold difference is reached such that the level ofscaling may be based on the difference between priority values. Itshould be understood that “scaling the priority values” or similarlanguage may include scaling actual media component values (e.g., audiolevels) and/or generating new values based on actual media componentvalues for the purpose of generating scaled haptic feedback based onmedia content.

For example, in a stereo embodiment involving audio content with a leftchannel and a right channel, if an excepted haptic effect is to beplayed at a right haptic output device, but not a left haptic outputdevice, the system can scale a portion of the audio content associatedwith the right channel in an upwards direction, and can scale a portionof the audio content associated with the left channel in a downwarddirection. The system can further generate a haptic effect based on thescaled portion of the audio content associated with the right channeland cause the haptic effect to be output at the right haptic outputdevice. Likewise, the system can further generate a haptic effect basedon the scaled portion of the audio content associated with the leftchannel and cause the haptic effect to be output at the left hapticoutput device. Thus, based on the dynamic scaling, the haptic effectthat is output at the right haptic output device can be more“pronounced” or “augmented,” whereas the haptic effect that is output atthe left haptic output device can be more “diminished.”

FIG. 1 illustrates a system 100 for providing haptic feedback based ondynamically scaled values of media components of media content 122,according to an embodiment of the invention. System 100 may include amedia source 120, a computing device 140, a haptic output device 162, animage output device 164, an audio output device 166, and/or othercomponents. Although only single instances of the aforementionedcomponents are illustrated in FIG. 1, more than one of such componentsmay be included. Media content 122 may include audio content, imagecontent (e.g., video, still images, visual representations of videogames, etc.), and/or other content. Examples of audio content caninclude 5.1 multi-channel audio, 6.1 multi-channel audio, 7.1multi-channel audio, 11.1 multi-channel audio, etc. Media content 122may be configured as video, audio, a video game representation, and/orother media that can be played or otherwise output.

In some embodiments, media source 120 may include an online mediaprovider, such as an online video provider or game server, a mediaplayer, such as a disc player or video game console, another computingdevice, and/or other device that can provide media 122 to computingdevice 140.

In some embodiments, computing device 140 may include one or moreprocessors 142 programmed with one or more computer program modules,which may be stored in one or more non-transitory storage media 144.Computing device 140 may be programmed by the one or more modules toprioritize media components of media content 122 and scale hapticfeedback based on the prioritization. The modules may include a mediaprocessing module 146, a prioritization module 148, a scaling module150, a haptic feedback module 152, a media output module 154, acoordination module 156, and/or other modules. In alternate embodiments,computing device 140 may be programmed by a single module, such as ahaptic spatialization module (not illustrated in FIG. 1), to prioritizemedia components of media content 122 and scale haptic feedback based onthe prioritization.

In some embodiments, media processing module 146 may be configured toobtain media content 122 from media source 120, computer readable media132, non-transitory storage media 144 (e.g., when media content 122 isstored locally at computing device 140), and/or other sources of mediacontent 122. For example, media content 122 may be streamed, downloaded,copied, stored locally, created locally (e.g., via a media capturedevice not illustrated in FIG. 1), or otherwise obtained by mediaprocessing module 146.

Whichever manner in which media content 122 is obtained, mediaprocessing module 146 may be configured to buffer media content 122 in amemory to process media content 122 into one or more media components,such as audio channels for audio content and image regions for imagecontent.

For example, media content 122 may include audio content that is encodedwith a stereo (i.e., dual channel) or surround (i.e., multi-channel)sound format. In these instances, media processing module 146 may decodemedia content 122 (e.g., audio portions thereof) into separate audiochannels using conventional stereo/multi-channel audio decodingtechniques used for delivering different audio channels to individualaudio output devices, such as speakers.

Media content 122 may additionally or alternatively include imagecontent that includes multiple streams (e.g., for separate displayscreens) and/or a single stream that can be divided into separateregions of the image. In these instances, media processing module 146may process media content 122 (e.g., image portions thereof) usingconventional image processing techniques, such as image segmentation tosegment an image into multiple regions. Each region may be grouped basedon clusters of pixels or grouped using other conventional imageprocessing techniques.

In some embodiments, prioritization module 148 may analyze individualmedia components and determine a priority value for each media componentbased on the analysis. The analysis may be performed in real-time suchthat as each media component is to be output by a media output device,the priority value of the media component may be determined.Alternatively, the analysis may be performed such that results of theanalysis are stored for later processing.

In embodiments where media content 122 includes audio content,prioritization module 148 may sample the audio content to determine anaudio characteristic of an audio channel. The audio characteristic mayinclude a magnitude of the audio and/or other characteristic of theaudio channel.

In some embodiments, prioritization module 148 may sample the audiocontent at a particular frame rate. Each frame of audio content may beprocessed such that the characteristic of the frame may be analyzed todetermine an audio characteristic of the audio frame. For each sampledportion (e.g., one or more frames) of the audio content, prioritizationmodule 148 may determine the audio characteristic for a given frame orother portion of the audio content. Processing the audio content may beimplemented using the system as disclosed in co-owned U.S. patentapplication Ser. No. 13/365,984, filed on Feb. 3, 2012, entitled “Soundto Haptic Effect Conversion System using Amplitude Value,” (hereinincorporated by reference in its entirety) and U.S. patent applicationSer. No. 13/366,010, filed on Feb. 3, 2012, entitled “Sound to HapticEffect Conversion System using Waveform,” (herein incorporated byreference in its entirety).

In some embodiments, prioritization module 148 may determine thepriority value based on the audio characteristic. For example, thepriority value may be equal to or otherwise derived from a value of theaudio characteristic. Differences of priority values in audioembodiments may indicate different levels, types, duration, etc., ofsound for different audio channels, which the system may use as thebasis for providing haptic feedback that varies spatially.

In embodiments where media content 122 includes image content,prioritization module 148 may determine a Hue Saturation Value (“HSV”)color histogram or other image processing values for a given segmentedregion of an image (e.g., a still image or a frame of a video image) anddetermine whether the HSV color histogram is different from one imageregion to another region. When separate video streams are used,prioritization module 148 may determine a HSV value or other valueobtained based on image processing for each stream.

The presence of a different object or differences in the scene may beindicated when the HSV color histogram differs from one region toanother or from one image stream to another image stream, which thesystem may use as the basis for providing haptic feedback that variesspatially. Other image variations may be used as the basis for providinghaptic feedback that varies spatially as well.

In some embodiments, prioritization module 148 may generate or otherwiseobtain a timestamp related to each frame and store the sampled mediacharacteristic in relation to the timestamp. In this manner, the mediacharacteristics may be stored in relation to the timestamps forcoordination of haptic feedback and output of media content 122.

In some embodiments, scaling module 150 may be configured to compare thepriority values determined for each of the media components anddetermine whether such values should be scaled based on the comparison.The comparison may be performed based on various techniques.

In some embodiments, for example, scaling module 150 may performpairwise comparisons of each of the priority values to determine whethera difference between a given pair of priority values exceeds apredetermined threshold value such that scaling should be performed. Theforegoing example may be useful when two media components are analyzed(such as for stereo audio embodiments), although such comparison may beapplied when other numbers of media components are analyzed as well. Theforegoing and other threshold values described herein may beconfigurable such that individual vendors/entities may adjust thethreshold values.

In some embodiments, scaling module 150 may determine an average ofpriority values for two or more of the media components and determinewhether a difference between a given priority value and the averageexceeds a predetermined threshold value (which may be the same ordifferent from the aforementioned threshold value). In some of theseembodiments, scaling module 150 may determine a standard deviation andperform various statistical techniques to determine whether the priorityvalues vary beyond a threshold variance such that scaling should beperformed. The foregoing example may be useful when a plurality of mediacomponents are analyzed (such as for surround audio embodiments),although such comparison may be applied when other numbers of mediacomponents are analyzed as well.

Regardless of how the priority values are compared, in some embodiments,scaling module 150 may scale one or more of the priority values. Forexample, scaling module 150 may generate a scaled value that is higheror lower than the original priority value. When the difference betweenthe priority values of a pair of priority values exceeds thepredetermined threshold value, scaling module 150 may scale up thehigher of the two priority values and/or scale down the lower of the twopriority values. Similarly, when the difference between a given priorityvalue and the average of a plurality of priority values exceeds thepredetermined threshold value, scaling module 150 may scale up the givenpriority value if the given priority value is higher than the averageand scale down the given priority value if the given priority value islower than the average. In some embodiments, scaling module 150 maydetermine baseline haptic feedbacks to provide based on non-scaledpriority values. In these embodiments, scaling module 150 may determinea difference between a characteristic of the haptic feedbacks (e.g.magnitudes, frequencies, durations, etc.) and scale the haptic feedbackcharacteristics with respect to one another.

Whichever embodiment of scaling is performed, scaling module 150 mayenhance the difference between two or more priority values that each isdetermined for a given media component. This allows a first hapticfeedback for a corresponding priority value to be provided at a firstspatial position that is more individually perceptible than a secondhaptic feedback that is provided at a second spatial position.

On the other hand, when the difference does not exceed the predeterminedthreshold, scaling module 150 may not scale the priority values. Thismay be the case when the difference in media components should notnecessarily be perceptibly different (e.g., when loud explosions areoccurring across different audio channels).

In some embodiments, haptic feedback module 152 may be configured togenerate control signals based on scaled priority values (for priorityvalues that are scaled) and/or the non-scaled priority values (forpriority values that are not scaled). For example, haptic feedbackmodule 152 may be configured to generate a first control signal based ona first priority value (which may or may not be scaled) and generate asecond control signal based on a second priority value (which may or maynot be scaled). Other control signals may be generated for otherpriority values corresponding to other media components as well. Thefirst control signal may be configured such that it causes a firsthaptic output device to provide a first haptic feedback. Likewise, thesecond control signal may be configured such that it causes a secondhaptic output device to provide a second haptic feedback.

As used herein with respect to control signals, the terms “first controlsignal” and “second control signal” (and similar language thatindividually identify control signals) are intended to convey thatdifferent haptic output devices may be caused to provide differenthaptic feedback. In some embodiments, for example, the “first controlsignal” and the “second control signal” may be separately provided tofirst and second haptic output devices, which each cause haptic feedbackto be provided based on their respective control signals. In someembodiments, the “first control signal” and the “second control signal”may be together provided to both first and second haptic output deviceswith appropriate addressing mechanisms such that the first and secondhaptic output devices decode the addressing mechanisms to determinewhich haptic feedback to output.

In some embodiments, the prioritized values may be stored in associationwith corresponding haptic feedback to be provided in a lookup tablestored in a memory, such as non-transitory storage media 144, and/or maybe dynamically determined based on priority values. For example, thepriority values may be used to determine a characteristic (e.g.,magnitude) of haptic feedback such that a higher priority value resultsin a higher magnitude haptic feedback.

In some embodiments, media output module 154 may be configured to outputmedia content 122. Media output module 152 may provide media 122 usingconventional media formats to, for example, image output device 164,audio output device 166, and/or other media output device.

In some embodiments, coordination module 156 may be configured tocoordinate media output and control signal output respectively to mediaoutput devices (e.g., image output device 164 and audio output device166) and haptic output devices 162. In some embodiments, coordinationmodule 156 may synchronize the image output, control signal (e.g.,haptic feedback caused by the control signal), and audio outputaccording to a synchronizing code embedded into media content 122 and/orvia time stamps that generally set a time for outputting each of thevideo output, audio output, and haptic feedback.

As would be appreciated, media content 122 may include both audiocontent and video content. Computing device 140 may process audio,video, and/or other media of media content 122 as described herein toprovide haptic feedback that varies spatially.

In some embodiments, haptic output device 162 may include an actuator,for example, an electromagnetic actuator, such as an Eccentric RotatingMass (“ERM”) in which an eccentric mass is moved by a motor, a LinearResonant Actuator (“LRA”) in which a mass attached to a spring is drivenback and forth, or a “smart material” such as piezoelectric,electro-active polymers, or shape memory alloys, a macro-composite fiberactuator, an electro-static actuator, an electro-tactile actuator,and/or another type of actuator that provides a physical feedback, suchas a haptic (e.g., vibrotactile) feedback. The haptic output device 162may include non-mechanical or non-vibratory devices, such as those thatuse electrostatic friction (“ESF”), ultrasonic surface friction (“USF”),or those that induce acoustic radiation pressure with an ultrasonichaptic transducer, or those that use a haptic substrate and a flexibleor deformable surface, or those that provide projected haptic output,such as a puff of air using an air jet, and so on.

In some embodiments, image output device 164 may include a display, suchas a touchscreen display, a monitor, and/or other display that candisplay the image content such as videos, still images, gamerepresentations, etc. In some embodiments, audio output device 166 mayinclude speakers, headphones, or other devices that can emit audio. Insome embodiments, image output device 164, haptic output device 162,and/or audio output device 166 may be integrated with computing device140. In some embodiments, image output device 164, haptic output device162, and/or audio output device 166 may be housed separately fromcomputing device 140.

In some embodiments, media source 120 may be programmed with the modulesdiscussed in relation to computing device 140. Thus, at least some orall of the functions described above with respect to computing device140 may be performed at media source 120.

Non-transitory storage media 144 may include one or both of systemstorage that is provided integrally (i.e., substantially non-removable)with computing device 140 and/or removable storage that is removablyconnectable to computing device 140 via, for example, a port (e.g., auniversal serial bus (“USB”) port, a firewire port, etc.) or a drive(e.g., a disk drive, etc.). Non-transitory storage media 144 may includeone or more of optically readable storage media (e.g., optical disks,etc.), magnetically readable storage media (e.g., magnetic tape,magnetic hard drive, floppy drive, etc.), electrical charge-basedstorage media (e.g., electronically erasable programmable read-onlymemory (“EEPROM”), random-access memory (“RAM”), etc.), solid-statestorage media (e.g., flash drive, etc.), and/or other electronicallyreadable storage media. Non-transitory storage media 144 may include oneor more virtual storage resources (e.g., cloud storage, a virtualprivate network, and/or other virtual storage resources). Non-transitorystorage media 144 may store software algorithms, information determinedby processor(s) 142, information received from computing device 140,and/or other information that enables computing device 140 to functionas described herein.

Processor(s) 142 are configured to provide information processingcapabilities in computing device 140. As such, processor(s) 142 mayinclude one or more of a digital processor, an analog processor, adigital circuit designed to process information, an analog circuitdesigned to process information, a state machine, and/or othermechanisms for electronically processing information. Althoughprocessor(s) 142 is illustrated in FIG. 1 as a single entity, this isfor illustrative purposes only. In some embodiments, processor(s) 142may include a plurality of processing units. These processing units maybe physically located within the same device, or processor(s) 142 mayrepresent processing functionality of a plurality of devices operatingin coordination. Processor(s) 142 may be configured to execute modulesby software; hardware; firmware; some combination of software, hardware,and/or firmware; and/or other mechanisms for configuring processingcapabilities on processor(s) 142.

The various modules described herein are example modules only. Otherconfigurations and numbers of modules may be used, as well usingnon-modular approaches so long as the one or more physical processorsare programmed to perform the functions described herein. It should beappreciated that although the various modules are illustrated in FIG. 1as being co-located within a single processing unit, in embodiments inwhich processor(s) 142 includes multiple processing units, one or moremodules may be located remotely from the other modules. The descriptionof the functionality provided by the different modules described hereinis for illustrative purposes, and is not intended to be limiting, as anyof modules may provide more or less functionality than is described. Forexample, one or more of the modules may be eliminated, and some or allof its functionality may be provided by other ones of the modules. Asanother example, processor(s) 142 may be configured to execute one ormore additional modules that may perform some or all of thefunctionality attributed herein to one of modules.

The components illustrated in FIG. 1 may be communicably coupled to oneanother via various communication links, such as a network. The networkmay include wired or wireless connections. In some aspects of theinvention, the network may include any one or more of, for instance, theInternet, an intranet, a Personal Area Network (“PAN”), a Local AreaNetwork (“LAN”), a Wide Area Network (“WAN”), a Storage Area Network(“SAN”), a Metropolitan Area Network (“MAN”), a wireless network, acellular communications network, a Public Switched Telephone Network(“PSTN”), and/or other network.

Various inputs, outputs, configurations, and/or other informationdescribed herein as being stored or storable may be stored in one ormore databases (not illustrated in FIG. 1). Such databases may be,include, or interface to, for example, an Oracle™ relational databasesold commercially by Oracle Corporation. Other databases, such asInformix™, Database 2 (“DB2”) or other data storage, includingfile-based, or query formats, platforms, or resources, such as On LineAnalytical Processing (“OLAP”), Standard Query Language (“SQL”), a SAN,Microsoft Access™ or others may also be used, incorporated, or accessed.The database may comprise one or more such databases that reside in oneor more physical devices and in one or more physical locations. Thedatabase may store a plurality of types of data and/or files andassociated data or file descriptions, administrative information, or anyother data.

Although not illustrated in FIG. 1, media content 122 may includedifferent sensor information. In some embodiments, the different sensorinformation may be associated with different objects on the image. Eachof the sensor information may be separated by media processing module146 and prioritized and scaled as described herein with respect to othermedia components. As such, computing device 140 may prioritize, scale,and provide dynamic haptic feedback on various types of informationincluded in media content 122.

FIG. 2 is a flow diagram 200 illustrating data flows between componentsin system 100 for providing haptic feedback based on dynamically scaledpriority values of media components of media content, according to anembodiment of the invention. Various haptic output devices 162(illustrated in FIG. 2 as haptic output devices 162A, 162B, . . . ,162N) may be collocated or otherwise associated with (illustrated inFIG. 2 as ellipses) corresponding media output devices 210 (illustratedin FIG. 2 as media output devices 210A, 210B, 210N). Media outputdevices 210 may include audio output devices, image output devices,and/or other types of media output device. Haptic output devices 162 maybe configured to output haptic feedback that is dynamically generatedand scaled based on media content 122. The haptic feedback may be outputin coordination with the output of media content 122 via haptic outputdevice 162, enriching the media experience for user(s).

In some embodiments, media processing module 146 may obtain mediacontent 122 and process the media content into a plurality (e.g., two ormore) of media components (illustrated in FIG. 2 as MC(A), MC(B), andMC(N)). Prioritization module 148 may determine priority values(illustrated in FIG. 2 as PV(A), PV(B), and PV(N)) for correspondingmedia components. Scaling module 150 may compare the priority values andscale each priority value to generate scaled priority values(illustrated in FIG. 2 as SV(A), SV(B), and SV(N)). In some embodiments,scaling module 150 may not perform scaling on one or more of thepriority values such that a given scaled priority value is equal to thenon-scaled priority value from prioritization module 150.

In some embodiments, scaling module 150 scales up or down a givenpriority value only when a difference between the given priority valueand another value exceeds a threshold value.

For example, when a difference between the given priority value andanother priority value exceeds a predetermined threshold, scaling module150 may scale the given priority value up or down and/or scale the otherpriority value up or down. In some embodiments, the given priority valueis scaled up and the other priority value is scaled down.

In another example, when a difference between the given priority valueand an average of priority values among a population of priority valuesexceeds a predetermined threshold value, scaling module 150 may scalethe given priority value up and/or down, depending on whether the givenvalue is above or below the average (e.g., the given priority value maybe scaled up when above the average and scaled down when below theaverage). Scaling module 150 may perform scaling based on other criteriaas well.

In yet another example, when a standard deviation among a population ofpriority values is or exceeds a predetermined threshold value, scalingmodule 150 may determine whether to perform scaling on the population ofpriority values, and then perform individualized scaling as describedherein.

Based on the scaled priority values (which may include non-scaledpriority values), haptic feedback module 152 may generate correspondingcontrol signals (illustrated in FIG. 2 as CS(A), CS(B), and CS(N)). Eachof the control signals may be configured to cause a corresponding hapticfeedback device 162 to output haptic feedback in coordination with mediaoutput by media output devices 210.

In other embodiments, scaling module 150 may perform scaling withoutrespect to whether the difference exceeds the predetermined threshold.In these embodiments, scaling module 150 may scale priority values whenthere exists a non-zero difference between the priority values.

FIG. 3A illustrates an example of a computing device 140A forprioritizing audio channels with integrated haptic output devices 162(illustrated in FIG. 3A as haptic output devices 162A, 162B, 162N) andaudio output devices 166 (illustrated in FIG. 3A as audio output devices166A, 166B, 166N), according to an embodiment of the invention. Althoughillustrated as having two haptic devices 162 (i.e., 162A and 162B) andtwo audio output devices 166 (i.e., 166A and 166B), computing device140A may include additional integrated haptic and audio output devices(i.e., 162N and 166N).

As illustrated, computing device 140A may be used for outputting stereoaudio and coordinating output of haptic feedback that is based onscaling priority values related to corresponding audio channels. Forexample, computing device 140A may determine “right” and “left” priorityvalues based on an analysis of “right” and “left” audio channels.Computing device 140A may cause “right” and “left” haptic feedbacks tobe output in conjunction with output of the right and left audiochannels through right and left speakers. The right haptic feedback maybe based on the right priority value and the left haptic feedback may bebased on the left priority value. As such, when the right and leftpriority values are different but similar to one another, right and lefthaptic feedback may be difficult to distinguish, particularly inembodiments where computing device 140A includes a handheld device. Byscaling the right priority values and/or the left priority values,computing device 140A may generate right and left haptic feedbacks thatare distinguishable from one another, thereby enhancing the dynamicallydetermined haptic feedbacks.

Computing device 140A may output the right and left audio channels vialeft audio output device 166A and right audio output device 166B andcoordinate such output with left and right haptic feedbacks via lefthaptic output device 162A and right haptic output device 162B. Becausethe left haptic feedback and/or the right haptic feedback is based onscaled priority values of the left and right audio channels, the usermay distinguish the left and right haptic feedbacks even if the left andright audio channels each include audio that is similar to one another.

FIG. 3B illustrates an example of a computing device 140B forprioritizing image components with integrated haptic output devices 162(illustrated in FIG. 3B as haptic output devices 162A, 162B, 162C, 162N)and image output device 164, according to an aspect of the invention. Asillustrated, image output device 164 may include a touch screen displayand/or other type of display that can output image content (e.g., video,still images, game representations, etc.). Image output device 164 maybe separated into display regions 310 (illustrated in FIG. 3B as displayregions 310A, 310B, 310C, 310N). The display regions 310 may beconceptual (e.g., software-based) or actual (e.g., include individualhardware displays).

Each of the display regions 310 may be configured to display acorresponding region of image content and be associated with acorresponding haptic output device 162 (i.e., one of haptic outputdevices 162A, 162B, 162C, 162N). Computing device 140B may process eachregion of image content using conventional image processing techniquesand determine corresponding priority values. Scaling and generation ofcontrol signals may be performed as described herein to provide distincthaptic feedbacks to haptic output devices 162.

FIG. 4 illustrates an example of a computing device 140 for prioritizingaudio channels with remote haptic output devices 162 (illustrated inFIG. 4 as haptic output devices 162A, 162B, 162N) and audio outputdevices 166 (illustrated in FIG. 4 as audio output devices 166A, 166B,166N), according to an embodiment of the invention. FIG. 4 furtherillustrates apparatus 410. Apparatus 410 may be separated intosoftware-derived or actual regions 412 (illustrated in FIG. 4 as regions412A, 412B, 412N). The relative positioning of regions 412 maycorrespond to the relative positioning of audio output devices 166.

In some embodiments, apparatus 410 may be configured as a chair havingdistinct regions, each having a corresponding haptic output device 162(i.e., one of haptic output devices 162A, 162B, 162N). Each of thedistinct regions may correspond to an audio output device 166 (i.e., oneof audio output devices 166A, 166B, 166N) such that computing device 140coordinates output of audio at a given audio output device 166 withhaptic feedback output to a corresponding haptic output device 162 at aposition in the chair that corresponds with the position of the givenaudio output device 166. For example, audio output at a “middle” audiooutput device 166 may be coordinated with haptic feedback provided via a“middle” haptic output device 162. FIG. 4 further illustrates apparatus420. Although illustrated in shadow outline, apparatus 420 may have aconfiguration that is similar to apparatus 410. Other numbers of hapticand audio output devices 166 may be used as well.

Although not illustrated, image content may be additionally oralternatively coordinated with apparatus 410 such that different imageregions may be prioritized and scaled, and have haptic feedback providedvia corresponding haptic output devices 162.

Referring to FIGS. 5A, 5B, 5C, and 5D (i.e., FIGS. 5A-D), theillustrated graphs plot a function 510 on a plot of level of scaling(illustrated as “SCALING”) versus the difference between a givenpriority value and another value (such as another priority value, avariance from an average, etc.) (illustrated as “PV DIFFERENCE”).

FIG. 5A is an example of a graph 500A that illustrates scaling ofpriority values that decreases as a difference between a priority valueand another value increases, according to an embodiment of theinvention. A computing device (e.g., computing device 140) may performscaling based on a function 510A. In some embodiments, the computingdevice may scale priority values without respect to whether a differencebetween priority values exceeds a predetermined threshold. For example,the computing device may scale priority values once a non-zerodifference has been determined and may decrease the level of scaling asthe difference increases. For these embodiments, as the differenceincreases, scaling may no longer be necessary because the priorityvalues may cause haptic feedback to be dynamically generated that issufficiently distinguishable from one another.

FIG. 5B is an example of a graph 500B that illustrates scaling ofpriority values that increases as a difference between a priority valueand another value increases, according to an embodiment of theinvention. A computing device (e.g., computing device 140) may performscaling based on a function 510B. In some embodiments, the computingdevice may scale priority values without respect to whether a differencebetween priority values exceeds a predetermined threshold. For example,the computing device may scale priority values once a non-zerodifference has been determined and may increase the level of scaling asthe difference increases. For these embodiments, as the differenceincreases, scaling may also be increased to further reinforcedifferences in audio, video, and/or other media content usingdynamically generated haptic feedback.

FIG. 5C is an example of a graph 500C that illustrates scaling ofpriority values that decreases as a difference between a priority valueand another value increases only after a threshold difference 520A hasbeen met, according to an embodiment of the invention. A computingdevice (e.g., computing device 140) may perform scaling based on afunction 510C. In some embodiments, the computing device may scalepriority values only after the difference between priority valuesexceeds a predetermined threshold value 520A and may increase the levelof scaling as the difference increases.

FIG. 5D is an example of a graph 500D that illustrates scaling ofpriority values that increases as a difference between a priority valueand another value increases only after a threshold difference 520B hasbeen met, according to an embodiment of the invention. A computingdevice (e.g., computing device 140) may perform scaling based on afunction 510D. In some embodiments, the computing device may scalepriority values only after the difference between priority valuesexceeds a predetermined threshold value 520B and may decrease the levelof scaling as the difference increases.

Although the functions illustrated in FIGS. 5A-5D are illustrated asnon-linear, other forms and shapes may be used as well. For example, forat least some portion of any of the foregoing functions, scaling may belinear such that an amount of scaling increases linearly with thedifference. Furthermore, a maximum and/or a minimum level of scaling maybound each of the functions such that scaling is capped or flooredaccording to particular needs. Each of the threshold values, functions,maximums, minimums, etc., may be configured to suit particular needs.

FIG. 6 illustrates an example of a process 600 for providing hapticfeedback based on dynamically scaled priority values of media componentsof media content, according to an embodiment of the invention. In oneembodiment, the functionality of the flow diagram of FIG. 6, as well asthe functionality of the flow diagrams of FIGS. 7, 8, and 9 are eachimplemented by software stored in memory or other computer readable ortangible medium, and executed by a processor. In other embodiments, eachfunctionality may be performed by hardware (e.g., through the use of anapplication specific integrated circuit (“ASIC”), a programmable gatearray (“PGA”), a field programmable gate array (“FPGA”), etc.), or anycombination of hardware and software.

The functionality depicted in the flowchart of FIGS. 6, 7, 8, and 9 aredescribed in greater detail herein. The described functionality may beaccomplished using some or all of the system components described indetail above. According to an aspect of the invention, the functionalitymay be performed in different sequences. In other embodiments,additional functionality may be performed along with some or all of thefunctionality shown in FIGS. 6, 7, 8 and 9, or some functionality may beomitted. In yet other embodiments, some or all of the functionality maybe performed simultaneously. Accordingly, the functionality asillustrated (and described in greater detail below) is an example offunctionality provided according to embodiments of the invention and, assuch, should not be viewed as limiting.

At 602, media content may be processed into a plurality of mediacomponents. For example, the media content may include audio content,image content, and/or other types of content. The audio content may beprocessed into separate audio channels and the image content may beprocessed into different regions of an image and/or different streams ofvideo (for example).

At 604, a priority value may be determined for each of the mediacomponents. The priority value may be determined based on variouscharacteristics of the media component (e.g., audio level for audio andbrightness level for images). At 606, the determined priority values maybe compared. Such comparisons may include pairwise comparisons,comparisons of a given priority value with an average of priority valuesfor the media components, and/or other comparisons.

At 608, control signals may be generated based on the comparisons. At610, each control signal may be provided to a corresponding hapticoutput device, where the control signal causes a haptic feedback to beoutput. At 612, the media content may be provided to media outputdevices, which may include audio output devices, image output device,and/or other media output devices.

FIG. 7 illustrates an example of scaling priority values, according toan embodiment of the invention. In some embodiments, 608 of process 600may include at least some of the following functionality based oncomparisons of 606.

At 702, a difference may be determined based on the comparisons. At 704,a determination of whether the difference exceeds (e.g., is greater thanor equal to) a predetermined threshold value may be made. If thedifference does not exceed the predetermined threshold value, thecontrol signals may be generated without scaling at 706. On the otherhand, if the difference exceeds the predetermined threshold value, oneor more of the priority values may be scaled at 708. The control signalsmay be generated based on the scaled priority values at 710.

FIG. 8 illustrates an example of scaling priority values, according toan embodiment of the invention. In some embodiments, 608 of process 600may include at least some of the following functionality based oncomparisons of 606. At 802, a difference may be determined based on thecomparisons. At 804, the priority values may be scaled based on thedifference without respect to whether such difference exceeds apredetermined threshold. At 806, the control signals may be generatedbased on the scaled priority values.

FIG. 9 illustrates an example of a process for providing haptic feedbackbased on input components of media content, according to an embodimentof the invention. The flow begins as input media content is processedinto n input components, where n can be any number. In the illustratedembodiment of FIG. 9, input audio content is sampled into n input audiostreams or channels. However, in alternate embodiments, input imagecontent can be sampled into n input video streams or channels. The flowthen proceeds to 910. At 910, one or more properties of the n inputcomponents are analyzed. Example properties can include a magnitude, aforce output, a density, a texture, or any other type of property. Inthe illustrated embodiment, where the n input components are n audioinput streams, a specific frequency range of each audio input streamsuch as a low frequency range, can be analyzed. The flow then proceedsto 920.

At 920, it is determined, for each time segment, which of the n inputcomponents have the highest priority by comparing priority values thatare determined for the n input components. A priority value can bedetermined for each input component of the n input components based onthe one or more analyzed properties of each input component. A priorityvalue for each input component can be compared with a priority value forat least one other input component (or an aggregate value, such as anaverage). In an embodiment, a comparative threshold (e.g., 40% or anyother percentage) can be defined, where the threshold may be required tobe met in order for adaptive scaling (described below in greater detailin conjunction with 930) to occur. For example, if priority values of aplurality of input components are compared and determined to be within40% of each other (i.e., do not meet the threshold), then the comparedinput components are considered to have the same priority, and noscaling is performed on the compared input components. However, ifpriority values of the input components are compared and determined notto be within 40% of each other (i.e., do meet the threshold), then thecompared input components are considered to have different priorities,and scaling can be performed on one or more of the compared inputcomponents. In an alternate embodiment, an absolute threshold, ratherthan a comparative threshold, can be used. In this alternate embodiment,priority values of the n input components can be compared with theabsolute threshold. If a priority value of an input component meets thethreshold, scaling can be performed on the input component. Otherwise,if the input component does not meet the threshold, no scaling isperformed on the input component. The flow then proceeds to 930.

At 930, one or more input components of the n input components arescaled based on the comparison of priority values of the n inputcomponents in 920. Input components that are determined to have a higherpriority can be scaled up, input components that are determined to havea lower priority can be scaled down, or a combination of the two. Byscaling an input component, according to an embodiment, one or moreproperties of the input component can be scaled. In an alternativeembodiment, information used to generate haptic feedback based on theinput component can be scaled, so that the input component is unaltered,and the haptic feedback is either stronger (when the information isscaled up) or weaker (when the information is scaled down). In yetanother alternate embodiment, a priority value can be scaled, and thepriority value can be used to generate the haptic feedback, where thehaptic feedback is also based on the input component. By scaling the oneor more input components, input components that are determined to have ahigher priority can gain clarity, and are less likely to be overshadowedby the other input components. In an alternate embodiment, rather thanscaling an input component, an input component can be modified (e.g.,frequency-shifted or pitch-shifted) or filtered. In certain embodiments,no input components are scaled, and 930 is omitted. The flow thenproceeds to 940.

At 940, the n input components are output to media output devices. Incertain embodiments, control signals are generated based on the n inputcomponents, and the control signals are output to haptic output devices,where each control signal causes haptic feedback to be output. In theillustrated embodiment, where the n input components are n audio inputstreams, the n audio input streams can be output to audio outputdevices. In certain embodiments, fewer than n input components areoutput to media output devices. In these embodiments, input componentsthat are determined to have a higher priority are output, and theremaining input components can be discarded. Further, in certainembodiments, a number of control signals that is fewer than n can begenerated and sent to a number of haptic output devices that is alsofewer than n. The flow then ends.

Thus, in one embodiment, a system can provide haptic feedback thatvaries spatially based on dynamically scaled values of media components.The system can scale one or more portions of the haptic feedback basedon an analysis of the media components. In this manner, the system canimprove the ability to distinguish between different haptic feedbackthat is provided in association with different media components and canprovide a richer, more immersive media experience. In other words, thesystem can aid in creating a more spatialized haptic experience byhighlighting and exaggerating difference in haptic feedback. Forexample, in a multi-channel experience, the system can distinguish ahaptic effect associated with a channel from the other haptic effectsassociated with the other channels. This can result in an idealmulti-channel experience, creating a spatialization of haptic effectsthat is pleasing to users.

The features, structures, or characteristics of the invention describedthroughout this specification may be combined in any suitable manner inone or more embodiments. For example, the usage of “one embodiment,”“some embodiments,” “certain embodiment,” “certain embodiments,” orother similar language, throughout this specification refers to the factthat a particular feature, structure, or characteristic described inconnection with the embodiment may be included in at least oneembodiment of the present invention. Thus, appearances of the phrases“one embodiment,” “some embodiments,” “a certain embodiment,” “certainembodiments,” or other similar language, throughout this specificationdo not necessarily all refer to the same group of embodiments, and thedescribed features, structures, or characteristics may be combined inany suitable manner in one or more embodiments.

One having ordinary skill in the art will readily understand that theinvention as discussed above may be practiced with steps in a differentorder, and/or with elements in configurations which are different thanthose which are disclosed. Therefore, although the invention has beendescribed based upon these preferred embodiments, it would be apparentto those of skill in the art that certain modifications, variations, andalternative constructions would be apparent, while remaining within thespirit and scope of the invention. In order to determine the metes andbounds of the invention, therefore, reference should be made to theappended claims.

We claim:
 1. A computer-readable medium having instructions storedthereon that, when executed by a processor, cause the processor toprovide haptic feedback based on media content, the providingcomprising: processing the media content into a plurality of componentscomprising a first component and a second component; determining a firstpriority value related to the first component and a second priorityvalue related to the second component; comparing the first priorityvalue with the second priority value; and generating a first controlsignal and a second control signal based on the comparison; wherein thefirst control signal is configured to cause a first haptic feedback tobe output and the second control signal is configured to cause a secondhaptic feedback to be output that is the same or different than thefirst haptic feedback.
 2. The computer-readable medium of claim 1, theproviding further comprising: providing the first control signal to afirst haptic output device, wherein the first control signal causes thefirst haptic output device to output the first haptic feedback; andproviding the second control signal to a second haptic output device,wherein the second control signal causes the second haptic output deviceto output the second haptic feedback.
 3. The computer-readable medium ofclaim 1, the providing further comprising: determining a differencebetween the first priority value and the second priority value based onthe comparison, wherein the first control signal and the second controlsignal are generated based on the determined difference.
 4. Thecomputer-readable medium of claim 3, the providing further comprising:determining that the difference does not exceed a threshold value; anddetermining the first haptic feedback based on a value of acharacteristic of the first component, wherein the first control signalis generated such that the first haptic feedback is caused to be output.5. The computer-readable medium of claim 3, the providing furthercomprising: determining that the difference exceeds a threshold value;determining a first baseline haptic feedback based on a first value of acharacteristic of the first component; and scaling the first baselinehaptic feedback to the first haptic feedback such that the first hapticfeedback is a scaled version of the first baseline haptic feedback basedon the determination that the difference exceeds the threshold value,wherein the first control signal is generated such that the first hapticfeedback is caused to be output.
 6. The computer-readable medium ofclaim 5, wherein the scaling the first baseline haptic feedback to thefirst haptic feedback further comprises altering at least one of: anumber; a magnitude; a frequency; or a duration of the first baselinehaptic feedback.
 7. The computer-readable medium of claim 5, theproviding further comprising: determining a second baseline hapticfeedback based on a second value of a characteristic of the secondcomponent, the characteristic of the second component being the same asthe characteristic of the first component; and scaling the secondbaseline haptic feedback to the second haptic feedback such that thesecond haptic feedback is a scaled version of the second baseline hapticfeedback based on the determination that the difference exceeds thethreshold value, wherein the second control signal is generated suchthat the second haptic feedback is caused to be output.
 8. Thecomputer-readable medium of claim 7, wherein the first baseline hapticfeedback is scaled up to the first haptic feedback and the secondbaseline haptic feedback is scaled down to the second haptic feedbacksuch that a level of the first haptic feedback is increased relative tothe first baseline haptic feedback and a level the second hapticfeedback is decreased relative to the second baseline haptic feedback.9. The computer-readable medium of claim 1, wherein the media contentcomprises audio content and the first component comprises a first audiochannel of the audio content and the second component comprises a secondaudio channel of the audio component.
 10. The computer-readable mediumof claim 9, the providing further comprising: determining a first audioproperty of the first audio channel, wherein the first priority value isbased on the first audio property.
 11. The computer-readable medium ofclaim 9, the providing further comprising: providing the first controlsignal to a first haptic output device, wherein the first control signalcauses the first haptic output device to output the first hapticfeedback; providing the first audio channel to a first audio outputdevice that causes first audio related to the first audio channel to beoutput; providing the second control signal to a second haptic outputdevice, wherein the second control signal causes the second hapticoutput device to output the second haptic feedback; and providing thesecond audio channel to a second audio output device that causes secondaudio related to the second audio channel to be output.
 12. Thecomputer-readable medium of claim 11, wherein the first control signaland the first audio channel are provided such that the first hapticfeedback and the first audio are caused to be synchronously output. 13.The computer-readable medium of claim 1, wherein the media contentcomprises image content and the first component comprises a first imagearea of the image content and the second component comprises a secondimage area of the image content.
 14. The computer-readable medium ofclaim 13, the providing further comprising: determining a first imageproperty of the first image area, wherein the first priority value isbased on the first image property.
 15. A computer-implemented method forproviding haptic feedback based on media content, thecomputer-implemented method comprising: processing the media contentinto a plurality of components comprising a first component and a secondcomponent; determining a first priority value related to the firstcomponent and a second priority value related to the second component;comparing the first priority value with the second priority value; andgenerating a first control signal and a second control signal based onthe comparison; wherein the first control signal is configured to causea first haptic feedback to be output and the second control signal isconfigured to cause a second haptic feedback to be output that is thesame or different than the first haptic feedback.
 16. Thecomputer-implemented method of claim 15, further comprising: providingthe first control signal to a first haptic output device, wherein thefirst control signal causes the first haptic output device to output thefirst haptic feedback; and providing the second control signal to asecond haptic output device, wherein the second control signal causesthe second haptic output device to output the second haptic feedback.17. The computer-implemented method of claim 15, further comprising:determining a difference between the first priority value and the secondpriority value based on the comparison, wherein the first control signaland the second control signal are generated based on the determineddifference.
 18. A system for providing haptic feedback based on mediacontent, the system comprising: a memory configured to store a hapticspatialization module; and a processor configured to execute the hapticspatialization module stored on the memory; wherein the processor, whenexecuting the haptic spatialization module, is configured to process themedia content into a plurality of components comprising a firstcomponent and a second component; wherein the processor, when executingthe haptic spatialization module, is further configured to determine afirst priority value related to the first component and a secondpriority value related to the second component; wherein the processor,when executing the haptic spatialization module, is further configuredto compare the first priority value with the second priority value;wherein the processor, when executing the haptic spatialization module,is further configured to generate a first control signal and a secondcontrol signal based on the comparison; and wherein the first controlsignal is configured to cause a first haptic feedback to be output andthe second control signal is configured to cause a second hapticfeedback to be output that is the same or different than the firsthaptic feedback.
 19. The system of claim 18, wherein the processor, whenexecuting the haptic spatialization module, is further configured toprovide the first control signal to a first haptic output device,wherein the first control signal causes the first haptic output deviceto output the first haptic feedback; and wherein the processor, whenexecuting the haptic spatialization module, is further configured toprovide the second control signal to a second haptic output device,wherein the second control signal causes the second haptic output deviceto output the second haptic feedback.
 20. The system of claim 18,further comprising: wherein the processor, when executing the hapticspatialization module, is further configured to determine a differencebetween the first priority value and the second priority value based onthe comparison, wherein the first control signal and the second controlsignal are generated based on the determined difference.